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The result matching section 60 calculates a similarity Di = D(Soj, Sij) 
between the matching result Soj and the matching result Sy of each of the 
reference images in the reference image matching result storage section 50, 
and extracts reference images in the descending order of the similarity Di of 
5 the matching result (step 103). The result of the extraction becomes, for 
example, as shown in FIG. 8. As reference images having a high possibility 
of being an image of the same object as the input image, Ri, R5 and R2 are 
obtained in the cited order. Finally, the reference images having high 
similarities are displayed (step 104). 

10 As the calculation method of the similarity Di(Soj, Sij) of the 

matching result, normalized correlation, rank_correlation, or the like can be 
used. The rank correlation is correlation of candidate precedence of the 
matching result. Denoting the candidate precedence of the matching result 
Soj of the input image by Aoj, it follows that Ao,2 = 1, Ao,s = 2 and Ao,3 = 3 in 

1 5 the case of the matching result shown in FIG. 7. Denoting candidate 

precedence of the matching result Sjj of each of the reference images by Aij, 
for example, the Spearman's rank correlation can be obtained according to 
the expression 1 - 6S j(A 0 j-Aij) 2 /{N(N 2 -l)}. 

In the similarity calculation, the similarities may be calculated after 

20 conducting variable conversion on the variables (such as Soj, Sij and Aoj, Ay). 
The similarities may be calculated by weighting variables with weights g(Aoj, 
Ay) based on the candidate precedence Aoj and/or Ay. For example, specific 
gravities of high precedence candidates become great by setting g(Aoj, Ay) = 
l/(Aoj+Ay) and replacing the similarities Soj and Sy respectively by 

25 Soj/(Aoj+Ay) and Sy /(Aoj+Ay). The similarities may be calculated with low 
precedence candidates excluded. 
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In the same way as the operation of the first embodiment, 
representative three-dimensional object models Cj (j = 1, 2, ... , N) as shown 
in FIG. 3 are stored in the representative three-dimensional object model 
storage section 20. Reference images Ri (i = 1, 2, ... , M) respectively of the 
5 objects as shown in FIG. 4 are stored in the reference image storage section 
70, A result (similarity) Sij of matching of each of the reference images Ri 
with the representative three-dimensional object model Cj as shown in FIG. 
5 is stored in the reference image matching result storage section 50. 

In addition, M reference three-dimensional object models Bi (i = 1, 
10 2, ... , M) associated with the reference image Ri are previously stored in the 
reference three-dimensional object model storage section 21 as shown in FIG. 
18. 

It is supposed that an input image I(u, v) as shown in FIG. 6 is 
obtained by the image input section 10 at the time of matching of the input 

1 5 image (step 100 in FIG. 17). According to the same processing as the 

operation in the first embodiment, Ri, R5 and R2 are obtained in the cited 
order as reference images having a high possibility of being an image of the 
same object as the input image as shown in FIG. 8 by the image generation 
section 30, the image matching section 40, and the result matching section 

20 60 (steps 101, 102 and 103). 

With respect to, for example, the reference images Ri, R5 and R2 
which are three high precedence candidates in the matching result obtained 
from the result matching section 60, the second image generation section 31 
acquires associated reference three-dimensional object models Bi, B5 and B2 

25 from the reference three-dimensional object model storage section 21, and 
generates comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are 
close in input condition such as the pose and illumination to the input image 
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obtained from the image input section 10 (step 111). The generation of the 
comparison images Hjk(u, v) is conducted by using a method similar to the 
step 101. In other words, the second image generation section 31 generates 
L comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are close in 
5 input condition such as the pose and illumination to the input image, with 
respect to the reference three-dimensional object models Bj (j = 1, 5, 2) in the 
reference three-dimensional object model storage section 21. The second 
image matching section 41 finds a similarity S(I, Hjk) between the input 
image I(u, v) and each comparison image Hjk(u, v), and finds a maximum 

10 similarity S 0 j = maxk S(I, Hjk) every model (step 112). 

The matching results become, for example, as shown in FIG. 19. If 
So,5>So,i>So,2, then Rs, Ri and R2 are obtained in the cited order as reference 
images having a high possibility of being an image of the same object as the 
input image. Finally, the reference images having high similarities are 

15 displayed (step 104). 

An image matching system according to a fourth embodiment of the 
present invention will now be described in detail with reference to FIGS. 20 
to 22. 

With reference to FIG. 20, an image matching system according to 
20 the fourth embodiment of the present invention includes an image input 
section 10, an image generation section 30, an image matching section 40, a 
result matching section 60, a second image generation section 31, a second 
image matching section 41, a result display section 80, a reference image 
storage section 70, a reference image registration section 75, a 
25 representative three-dimensional object model storage section 20, a three- 
dimensional object model registration section 25, a reference image 
matching result storage section 50, a reference image matching result 
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in FIG. 3 are stored in the representative three-dimensional object model 
storage section 20. Reference images Ri (i = 1, 2, ... , M) respectively of the 
objects as shown in FIG. 4 are stored in the reference image storage section 
70. A result (similarity) Sij of matching of each reference image Ri with the 
5 representative three-dimensional object model Cj as shown in FIG. 5 is 
stored in the reference image matching result storage section 50. 

In addition, M reference three-dimensional object models Bi (i = 1, 
2, ... , M) associated with the reference image Ri are previously stored in the 
reference three-dimensional object model storage section 21 by the 
10 processing conducted at the time of reference image registration, as shown 
in FIG. 18. 

It is supposed that an input image I(u, v) as shown in FIG. 6 is 
obtained by the image input section 10 at the time of matching of the input 
image (step 100 in FIG. 17). According to the same processing as the 

1 5 operation in the first embodiment, Ri, Rs and R2 are obtained in the cited 
order as reference images having a high possibility of being an image of the 
same object as the input image as shown in FIG. 8 by the image generation 
section 30, the image matching section 40, and the result matching section 
60 (steps 101, 102 and 103). 

20 With respect to, for example, the reference images Ri, R5 and R2 

which are three high precedence candidates in the matching result obtained 
from the result matching section 60, the second image generation section 31 
acquires associated reference three-dimensional object models Bi, Bs and B2 
from the reference three-dimensional object model storage section 21, and 

25 generates comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are 

close in input condition such as the pose and illumination to the input image 
obtained from the image input section 10 (step 111). The second image 



42 



the pose condition) the same on the basis of each reference three- 
dimensional object model associated with the reference image obtained from 
the reference three-dimensional object model storage section 21. Thus, the 
image conversion section 36 generates partial images. The image conversion 
5 section 36 is similar to the image conversion section 35 in the second 
conventional technique shown in FIG. 28. 

The partial image matching section 45 conducts comparison on the 
partial images of the converted input image and reference image obtained 
from the image conversion section 36, and calculates the similarity. The 

10 similarity calculation is conducted in the same way as the step 102. 

General operation of the fifth embodiment will now be described in 
detail with reference to FIG. 23 and a flow chart shown in FIG. 24. 

At the time of input image matching, steps 100, 101, 102 and 103 
shown in FIG. 24 are the same as the operation conducted in the first 

1 5 embodiment shown in FIG. 2. With respect to a reference image of a high 
precedence candidate in the matching result obtained from the result 
obtained from the result matching section 60, the image conversion section 
36 converts the input image and/or the reference image so as to make the 
input condition (such as the pose condition) the same on the basis of each 

20 reference three-dimensional object model associated with the reference 
image obtained from the reference three-dimensional object model storage 
section 21. Thus, the image conversion section 36 generates partial images 
(step 121). 

The partial image matching section 45 conducts comparison on the 
25 partial images of the converted input image and reference image obtained 
from the image conversion section 36, and calculates the similarity (step 
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an image matching step of calculating a similarity between the input 
image and each of the comparison images generated at the image generation 
step, selecting a maximum similarity with respect to comparison images 
associated with each representative three-dimensional object model, and 
5 regarding the maximum similarity as a similarity between the input image 
and the representative three-dimensional object model; 

a step of storing the reference images of objects in a reference image 
storage section; 

a step of storing similarities between the reference images stored in 
10 the reference image storage section and representative three-dimensional 
object models stored in the representative three-dimensional object model 
storage section, in a reference image matching result storage section; 

a result matching step of extracting the reference image similar to 
the input image on the basis of similarities between the input image and the 
1 5 representative three-dimensional object models calculated at the image 
matching step and similarities between the reference images and the 
representative three-dimensional object models stored in the reference 
image matching result storage section; 

a step of storing reference three-dimensional object models 
20 associated with the reference images stored in the reference image storage 
section, in a reference three-dimensional object model storage section; 

an image conversion step of obtaining reference three-dimensional 
object models associated with reference images extracted at the result 
matching step, from the reference three-dimensional object model storage 
25 section, squaring an input condition of the input image with that of the 
reference image extracted at the result matching step by converting the 
input image and/or the reference image extracted at the result matching 
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result update step, for generating the reference three-dimensional object 
model associated with the reference image by combining the representative 
three-dimensional object models stored in the representative three- 
dimensional object model storage section on the basis of the similarity, and 
5 registering the generated reference three-dimensional object model in the 
reference three-dimensional object model storage section. 

35. The image matching method according to claim 34, wherein 
at the three-dimensional object model generation step, a reference 

three-dimensional object model associated with each reference image is 
1 0 generated by combining representative three-dimensional object models 
stored in the representative three-dimensional object model storage section 
every partial region, on the basis of similarities obtained every partial 
region between reference images stored in the reference image storage 
section and representative three-dimensional object models stored in the 
15 representative three-dimensional object model storage section, and the 
generated reference three-dimensional object model is registered in the 
reference three-dimensional object model storage section. 

36. The image matching method according to claim 33, wherein 
at the image matching step, a similarity between the input image 

20 and a representative three-dimensional object model is calculated every 
partial region, 

the reference image matching result storage section stores 
similarities between the reference images stored in the reference image 
storage section and representative three-dimensional object models stored 
25 in the representative three-dimensional object model storage section, every 
partial region, and 
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model associated with the reference image by combining the representative 
three-dimensional object models stored in the representative three- 
dimensional object model storage section on the basis of the similarity, and 
registering the generated reference three-dimensional object model in the 
5 reference three-dimensional object model storage section. 

54. The image matching program according to claim 53, wherein 
at the three-dimensional object model generation step, a reference 

three-dimensional object model associated with each reference image is 
generated by combining representative three-dimensional object models 

10 stored in the representative three-dimensional object model storage section 
every partial region, on the basis of similarities obtained every partial 
region between reference images stored in the reference image storage 
section and representative three-dimensional object models stored in the 
representative three-dimensional object model storage section, and the 

1 5 generated reference three-dimensional object model is registered in the 
reference three-dimensional object model storage section. 

55. The image matching program according to claim 52, wherein 
at the image matching step, a similarity between the input image 

and a representative three-dimensional object model is calculated every 
20 partial region, 

the reference image matching result storage section stores 
similarities between the reference images stored in the reference image 
storage section and representative three-dimensional object models stored 
in the representative three-dimensional object model storage section, every 
25 partial region, and 

at the result matching step, the reference image similar to the input 
image is extracted on the basis of similarities between the input image and 
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